Determining a target group based on product-specific affinity attributes and corresponding weights

ABSTRACT

A campaign profile specifies products and/or content items associated with a campaign. A target group selection engine applies an affinity attribute model to user information of a user. The affinity attribute model is used to determine the user&#39;s affinity towards (a) product attributes of the products associated with the campaign and/or (b) content attributes of the content items associated with the campaign. The affinity attribute model may be generated using machine learning. A user interface accepts target user tuning parameters that specify weights to be applied to the affinity attributes determined by the affinity attribute model. Based at least on applying the weights to the affinity attributes, an inclusion score and/or exclusion score for the user is determined. The user is included in a target group, for engaging with the campaign, based on the inclusion score and/or exclusion score.

INCORPORATION BY REFERENCE; DISCLAIMER

The following application is hereby incorporated by reference:application Ser. No. 16/666,070 filed on Oct. 18, 2019. The applicanthereby rescinds any disclaimer of claims scope in the parentapplication(s) or the prosecution history thereof and advise the USPTOthat the claims in the application may be broader that any claim in theparent application(s).

TECHNICAL FIELD

The present disclosure relates to determining a target group of users.In particular, the present disclosure relates to determining a targetgroup of users based on product-specific affinity attributes andcorresponding weights.

BACKGROUND

Campaign management applications allow for campaign content creation,campaign content dissemination, and/or campaign content responsetracking. Campaign management applications allow for specification of atarget audience for campaign content based on a segmentation approach.According to the segmentation approach, users are segmented based oncertain user attributes, such as age, gender, geographical location,interaction history. A campaign administrator selects certain segmentsof users to be included in the target audience. As an example, acampaign administrator may select users who are between 18-25 and femaleto be included. All other users would be excluded.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not by way oflimitation in the figures of the accompanying drawings. It should benoted that references to “an” or “one” embodiment in this disclosure arenot necessarily to the same embodiment, and they mean at least one. Inthe drawings:

FIG. 1 illustrates a target group selection system, in accordance withone or more embodiments;

FIG. 2 illustrates additional details of a target group selectionsystem, in accordance with one or more embodiments;

FIG. 3 illustrates a machine learning system for generating an inferreduser attribute model, in accordance with one or more embodiments;

FIG. 4 illustrates a machine learning system for generating an affinityattribute model, in accordance with one or more embodiments;

FIG. 5 illustrates an example set of operations for determining a targetgroup of users for engaging with a campaign, in accordance with one ormore embodiments;

FIGS. 6A-B illustrate example sets of operations for determining aninclusion group of users and an exclusion group of users, in accordancewith one or more embodiments;

FIG. 7 illustrates an example set of operations for using machinelearning to generate models for generating and/or enhancing userprofiles, in accordance with one or more embodiments;

FIGS. 8A-B illustrate an example user interface for specifying targetuser tuning parameters, in accordance with one or more embodiments;

FIGS. 9A-C shows example Venn diagrams illustrating inclusion groups,exclusion groups, and target groups, in accordance with one or moreembodiments;

FIG. 10 shows a block diagram that illustrates a computer system inaccordance with one or more embodiments.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding. One or more embodiments may be practiced without thesespecific details. Features described in one embodiment may be combinedwith features described in a different embodiment. In some examples,well-known structures and devices are described with reference to ablock diagram form in order to avoid unnecessarily obscuring the presentinvention.

-   -   1. GENERAL OVERVIEW    -   2. TARGET GROUP SELECTION SYSTEM ARCHITECTURE    -   3. MACHINE LEARNING SYSTEMS    -   4. DETERMINING A TARGET GROUP OF USERS FOR ENGAGING WITH A        CAMPAIGN    -   5. USING MACHINE LEARNING TO GENERATE MODELS FOR GENERATING        AND/OR ENHANCING USER PROFILES    -   6. EXAMPLE EMBODIMENTS    -   7. HARDWARE OVERVIEW    -   8. MISCELLANEOUS; EXTENSIONS

1. General Overview

One or more embodiments include determining a target group for engagingwith a campaign based on product-specific affinity attributes. Anaffinity attribute model is configured to determine an affinity of auser towards a product attribute. The term “affinity” may refer topositive affinity or negative affinity. A positive affinity represents alevel that a user favors or is attracted towards a product attribute. Anegative affinity represents a level that a user disfavors a productattribute. An affinity attribute model may be generated using machinelearning based on a training set of user information. Affinityattributes are “product-specific” if the affinity attributes aredetermined specifically with respect to product attributes of one ormore products of interest. The products of interest may be, for example,products that are being promoted and/or discounted in a particular salescampaign. A campaign management application may utilize an affinityattribute model to determine a target group of users based onproduct-specific affinity attributes. The campaign managementapplication may direct campaign content about the products of interestand/or sales campaign towards the target group of users, without sendingthe campaign content to other users. Campaign content may be moreeffective for the target group of users who have positive affinitiestowards product attributes of the products of interest, than for otherusers. For example, the target group of users may have a greaterlikelihood of interacting with the campaign content, or even making apurchase in response to the campaign content, than the other users.Hence, the campaign management application is able to take aproduct-specific approach that considers user affinities whendetermining a target group.

One or more embodiments include determining a target group for engagingwith a campaign based on utilizing different weights for affinityattributes, confirmable user attributes, and/or inferred userattributes. Users are associated with various user attributes, such asage, gender, geographical location, employment status, and the affinityattributes described above. Different user attributes have differentlevels of significance and/or correlations towards the effectiveness ofcampaign content for a particular user. As an example, users who areparents may be more likely to engage with a back-to-school campaign,whereas users who are retirees may be more likely to engage with asummer vacations promotion. Meanwhile, users who are not parents do notnecessarily have no interest in the back-to-school campaign. Users who,for example, live nearby the store and have positive affinity towardscraft supplies may still be likely to engage with the back-to-schoolcampaign. A user interface of a campaign management application isconfigured to accept tuning parameters specifying weights applicable tovarious user attributes. The campaign management application generatesan inclusion score function using the specified weights. The campaignmanagement application inputs affinity attributes, confirmable userattributes, and/or inferred user attributes of a user into the inclusionscore function, applies the specified weights to the user attributes,and thereby determines an inclusion score. The campaign managementapplication determines whether the user is included in the target groupbased on the inclusion score. Hence, the campaign management applicationis able to take an individualized approach that considers a broadspectrum of user attributes when determining a target group.

One or more embodiments described in this Specification and/or recitedin the claims may not be included in this General Overview section.

2. Target Group Selection System Architecture

FIG. 1 illustrates a target group selection system, in accordance withone or more embodiments. As illustrated in FIG. 1, a system 100 includesa target group engine 102, which includes an inclusion groupdeterminator 104 and an exclusion group determinator 106, and a targetgroup 108. In one or more embodiments, the system 100 may include moreor fewer components than the components illustrated in FIG. 1. Thecomponents illustrated in FIG. 1 may be local to or remote from eachother. The components illustrated in FIG. 1 may be implemented insoftware and/or hardware. Each component may be distributed overmultiple applications and/or machines. Multiple components may becombined into one application and/or machine. Operations described withrespect to one component may instead be performed by another component.

In one or more embodiments, a target group 108 includes a group of userswho are selected for engaging with a campaign. Engaging with a campaignincludes receiving one or more campaign content items associated withthe campaign. Campaign content includes content that promotes, markets,and/or otherwise describes, products and/or services associated with acampaign. Campaign content may be disseminated in any form, such asemails, video content, audio content, and/or physical mail.

A target group 108 may selected based on an inference and/or otherdetermination that the campaign content items will be more effective forthe target group 108, than for other users who are not within the targetgroup 108. For example, the target group of users may have a greaterlikelihood of engaging or interacting with the campaign content, or evenmaking a purchase in response to the campaign content, than the otherusers.

In one or more embodiments, a target group engine 102 refers to hardwareand/or software configured to perform operations described herein fordetermining a target group 108 of users for engaging with a campaign.Examples of operations for determining a target group 108 of users forengaging with a campaign are described below with reference to FIGS.5-6B.

In an embodiment, a target group engine 102 is implemented on one ormore digital devices. The term “digital device” generally refers to anyhardware device that includes a processor. A digital device may refer toa physical device executing an application or a virtual machine.Examples of digital devices include a computer, a tablet, a laptop, adesktop, a netbook, a server, a web server, a network policy server, aproxy server, a generic machine, a function-specific hardware device, amainframe, a television, a content receiver, a set-top box, a printer, amobile handset, a smartphone, and/or a personal digital assistant (PDA).

In one or more embodiments, a target group engine 102 includes aninclusion group determinator 104 and an exclusion group determinator106. An inclusion group determinator 104 refers to hardware and/orsoftware configured to determine an inclusion group of users. Anexclusion group determinator 106 refers to hardware and/or softwareconfigured to determine an exclusion group of users. A target group 108of users includes the inclusion group of users, while excluding theexclusion group of users. Hence, a user who is in the inclusion groupbut not in the exclusion group is a member of the target group 108. Auser who is in the inclusion group and in the exclusion group is not amember of the target group 108. A user who is not in the inclusion groupis not a member of the target group 108 (regardless of whether the useris in the exclusion group). Further details regarding an inclusion groupdeterminator 104 and an exclusion group determinator 106 are describedbelow with reference to FIG. 2.

FIG. 2 illustrates additional details of a target group selectionsystem, in accordance with one or more embodiments. As illustrated inFIG. 2, a system includes a user interface 200, an inclusion groupdeterminator 204, an inclusion group 222, an exclusion groupdeterminator 206, an exclusion group 224, and a data repository 230. Inone or more embodiments, the system may include more or fewer componentsthan the components illustrated in FIG. 2. The components illustrated inFIG. 2 may be local to or remote from each other. The componentsillustrated in FIG. 2 may be implemented in software and/or hardware.Each component may be distributed over multiple applications and/ormachines. Multiple components may be combined into one applicationand/or machine. Operations described with respect to one component mayinstead be performed by another component.

In one or more embodiments, as described above, an inclusion groupdeterminator 204 refers to hardware and/or software configured todetermine an inclusion group 222 of users. Examples of operations fordetermining an inclusion group 222 are described below with reference toFIG. 6A.

An inclusion group determinator 204 includes an inclusion score functionfor determining an inclusion score for each user. In an embodiment, aninclusion score function includes a set of inclusion ranges 214respectively applied to a set of user attributes (confirmable userattributes 234, inferred user attributes 236, and/or affinity attributes238) of a user. The inclusion score function determines whether one ormore user attributes are within corresponding inclusion ranges 214. Ifyes, then the user is included in an inclusion group 222. The inclusionscore function may output an inclusion score for the user that is abovea threshold value for including the user in the inclusion group 222.Otherwise, the user is excluded from the inclusion group 222. Theinclusion score function may output an inclusion score for the user thatis below a threshold value for including the user in the inclusion group222.

In an embodiment, an inclusion score function includes a set ofinclusion weights 214 respectively applied to a set of user attributes(confirmable user attributes 234, inferred user attributes 236, and/oraffinity attributes 238) of a user. The inclusion score functioncomputes a weighted sum of the user attributes. An example inclusionscore function is as follows:

Inclusion Score=w ₁ a ₁ +w ₂ a ₂ +w ₃ a ₃+ . . .

wherein w_(x) represent the inclusion weights 214, and a_(x) representthe user attributes. The inclusion score function determines whether theweighted sum is above a threshold value. If yes, then the user isincluded in an inclusion group 222. Otherwise, the user is excluded fromthe inclusion group 222.

In one or more embodiments, as described above, an exclusion groupdeterminator 206 refers to hardware and/or software configured todetermine an exclusion group 224 of users. Examples of operations fordetermining an exclusion group 224 are described below with reference toFIG. 6B.

An exclusion group determinator 206 includes an exclusion score functionfor determining an exclusion score for each user. An exclusion scorefunction is similar to an inclusion score function, as described above.In an embodiment, an exclusion score function includes a set ofexclusion ranges 218 respectively applied to a set of user attributes(confirmable user attributes 234, inferred user attributes 236, and/oraffinity attributes 238) of a user. In an embodiment, an exclusion scorefunction includes a set of exclusion weights 220 respectively applied toa set of user attributes (confirmable user attributes 234, inferred userattributes 236, and/or affinity attributes 238) of a user.

In an embodiment, an inclusion group determinator 204 and/or exclusiongroup determinator 206 are implemented on one or more digital devices.

In one or more embodiments, a data repository 230 is any type of storageunit and/or device (e.g., a file system, database, collection of tables,or any other storage mechanism) for storing data. Further, a datarepository 230 may include multiple different storage units and/ordevices. The multiple different storage units and/or devices may or maynot be of the same type or located at the same physical site. Further, adata repository 230 may be implemented or executed on the same computingsystem as an inclusion group determinator 204, exclusion groupdeterminator 206, and/or target group engine. Alternatively oradditionally, a data repository 230 may be implemented or executed on acomputing system separate from an inclusion group determinator 204,exclusion group determinator 206, and/or target group engine. The datarepository 104 may be communicatively coupled to the inclusion groupdeterminator 204, exclusion group determinator 206, and/or target groupengine via a direct connection or via a network.

Information describing user profiles 232, campaign profiles 240, andproduct profiles 246 may be implemented across any of components withinthe system. However, this information is illustrated within the datarepository 230 for purposes of clarity and explanation.

In one or more embodiments, a user profile 232 includes a set of userattributes derived from user information that may be collected from oneor more data sources. Data sources may include cookies associated with auser, web applications with personal information entered and/or sharedby a user, and public data sources with public information associatedwith a user.

A user profile 232 may include confirmable user attributes 234.Confirmable user attributes 234 are attributes that are directlyconfirmable based on user information collected from one or more datasources. As an example, a user may publicly post a resume on a socialmedia platform. The resume may provide the user's work history andeducational history. Hence, confirmable user attributes for the user mayinclude the companies the user has worked at, the titles he has held,the degrees he has obtained, the schools he has attended. Additionally,a government database of registered businesses may include addresses ofthe companies the user has worked at. Hence, confirmable user attributesfor the user may also include geographical locations associated with theuser.

A user profile 236 may include inferred user attributes 236. Inferreduser attributes 236 are attributes that are inferred from userinformation collected from one or more data sources. As an example,cookies of a user may indicate that the user has a recent web history ofsearching for maternity clothing. Based on the interaction history, aninferred user attribute may include that the user is an expectantmother. As another example, a ticketing application may indicate thatthe user has purchased ten opera show tickets in the past twelve months.Based on the purchase history, an inferred user attribute may includethat the user is an opera lover. In an embodiment, inferred userattributes 236 may be determined by an inferred user attribute modelgenerated using machine learning, which is further described below withreference to FIG. 3.

An inferred user attribute 236 may be represented as a score indicatinga level of confidence of the inference. Additionally or alternatively,an inferred user attribute 236 may be represented as a score indicatinga level at which a user exhibits the inferred user attribute 236. As anexample, Jane and Mary may both conduct online research on an academicplatform about solar power. Jane may view more academic articles thanMary. An inferred user attribute indicating that Jane is a Doctor ofPhilosophy (PhD) candidate in solar power may be associated with ahigher score than an inferred user attribute indicating that Mary is aPhD candidate in solar power.

A user profile 236 may include affinity attributes 238. An affinityattribute 238 are attributes indicating how much a user favors ordisfavors a certain attribute of a product and/or a campaign. Affinityattributes 238 may be inferred from user information collected from oneor more data sources. As an example, a shopping history of a user mayindicate that 75% of the clothing purchased has a dominant color ofblack. Based on the shopping history, an affinity attribute may includethat the user has a positive affinity towards black clothing. As anotherexample, a shopping history of a user may indicate that 2% of theclothing purchased has a dominant color of red. Based on the shoppinghistory, an affinity attribute may include that the user has a negativeaffinity towards red clothing. As another example, a shopping history ofa user may indicate that 70% of a user's clothing purchase are onclearance. Based on the shopping history, an affinity attribute mayinclude that the user has a positive affinity towards clearancecampaigns and clearance messaging. In an embodiment, affinity attributes238 may be determined by an affinity attribute model generated usingmachine learning, which is further described below with reference toFIG. 4.

An affinity attribute 238 may be expressed as a score indicating a levelof confidence of the determination of the affinity attribute 238.Additionally or alternatively, an affinity attribute 238 may beexpressed as a score indicating a level at which a user favors ordisfavors a certain attribute. As an example, Jane and Mary may bothshop for clothing online. 80% of clothing viewed by Jane may includepolka dot patterns. 70% of clothing viewed by Mary may include polka dotpatterns. Hence, Jane's positive affinity towards polka dot patterns maybe determined to be higher than that of Mary's.

In one or more embodiments, a campaign profile 240 includes informationdefining a sales campaign. A campaign administrator may specify theinformation via a user interface. Additionally or alternatively, anotherapplication may supply the information via an application programminginterface (API). As an example, an inventory application may supplyinformation regarding overstocked products. Based on the inventoryapplication, a campaign profile may specify the overstocked products asproducts to be promoted in a campaign.

A campaign profile 240 specifies a set of one or more products and/orservices 242 that are being promoted, discounted, and/or otherwisehighlighted by a campaign. As an example, products of a back-to-schoolcampaign may include backpacks, stationary, and school clothing. Asanother example, products of an end-of-summer campaign may includeshort-sleeve shirts, shorts, and swimsuits.

A product and/or service 242 is associated with a set of productattributes 248. Examples of product attributes 248 include a color,pattern, product type, dimensions of the product, specifications of theproduct, and product ratings.

A product profile 246 stores product attributes 248 of a particularproduct 242. A product profile 246 may be generated based on user inputand/or by another application. An application may analyze a product 242to determine associated product attributes 248. As an example, asoftware product may be tested to determine a throughput of the softwareproduct. The throughput may be stored as a product attribute. As anotherexample, a photo of a product may be analyzed to determine a color ofthe product. The color may be stored as a product attribute. As anotherexample, a label of a product may be analyzed using optical characterrecognition (OCR). A number of calories per serving of the product, asindicated on the label, may be stored as a product attribute.

A campaign profile 240 specifies a set of one or more content items 244.A content item 244 is an object and/or message to be transmitted to oneor more users to promote or otherwise bring attention to a campaignand/or associated products 242. A content item 244 included in acampaign profile 240 may be a draft of an object to be transmitted tousers or a final version of an object to be transmitted to users. Acontent item 244 may be disseminated in any form, such as emails, videocontent, audio content, and/or physical mail. Content items 244 directedtowards a target group 108 may be referred to as “campaign content.”

A content item 244 is associated with a set of content attributes.Content attributes may be determined based on user input and/or byanother application. An application may analyze a content item 244 todetermine associated content attributes. As an example, an applicationmay execute natural language processing on a content item to determine atheme of the content item. The theme may be stored as a contentattribute.

A campaign profile 240 specifies a set of one or more campaignattributes 245. Examples of campaign attributes 245 include a campaigntype, campaign duration, and promotion type. A campaign type may beseasonal or otherwise related to a time of year. Additionally oralternatively, a campaign type may be related to an inventory status.Examples of campaign types include “Back to School Promotion,”“Thanksgiving Sale,” “Winter Promotion,” “Clearance,” “New Arrivals.”Examples of promotion types include “Discounts on All Items,” “Discountson Marked-Down Items,” “Buy One Get One Free,” “Buy One Get One 50%Off.”

In one or more embodiments, as described above, a product and/or service242 is associated with a set of product attributes 248. Productattributes 248 are organized in a product attribute hierarchy. Theproduct attribute hierarchy is stored separately from product profiles236 in a data repository 230. In the product attribute hierarchy, aparent node may be associated with a particular product attribute, whilea child node may be associated with a sub-type of the particular productattribute. As an example, a product attribute hierarchy may include atop node associated with a product type, “clothing.” Two child nodes ofthe “clothing” node may be respectively associated with the producttypes, “tops” and “bottoms.” Two child nodes of the “bottoms” node maybe respectively associated with the styles, “solid colors” and“patterned.” Two child nodes of the “patterned” node may be respectivelyassociated with the patterns, “striped pattern” and “polka dot pattern.”

A particular product that is associated with a particular productattribute represented by a child node is considered to be alsoassociated with all product attributes represented by any ancestor nodesof the child node. Referring to the example product attribute hierarchyabove, a product that is associated with “patterned” is also consideredto be associated with “bottoms” (parent of “patterned”) and “clothing”(grandparent of “patterned”). However, the product is not necessarilyassociated with “striped pattern” or “polka dot pattern” (children of“patterned”).

In one or more embodiments, a user interface 200 refers to hardwareand/or software configured to facilitate communications between a userand an inclusion group determinator 204, exclusion group determinator206, and/or target group engine. A user interface 200 renders userinterface elements to present information to a user and/or to receiveuser input from a user. Examples of interfaces include a graphical userinterface (GUI), a command line interface (CLI), a haptic interface, anda voice command interface. Examples of user interface elements includecheckboxes, radio buttons, dropdown lists, list boxes, buttons, toggles,text fields, date and time selectors, command lines, sliders, pages, andforms.

In one or more embodiments, a user interface 200 accepts user inputspecifying target user tuning parameters 212. Target user tuningparameters 212 are parameters used to tune an inclusion score functionfor determining an inclusion group 222 and/or an exclusion scorefunction for determining an exclusion group 224. As described above, aninclusion score function includes one or more inclusion ranges 214and/or inclusion weights 216. An exclusion score function includes oneor more exclusion ranges 218 and/or exclusion weights 220.

3. Machine Learning Systems

FIG. 3 illustrates a machine learning system for generating an inferreduser attribute model, in accordance with one or more embodiments. Asillustrated in FIG. 3, a system includes user information 330, aninferred user attribute model 332, a machine learning algorithm 334, andone or more inferred user attributes 336. In one or more embodiments,the system may include more or fewer components than the componentsillustrated in FIG. 3. The components illustrated in FIG. 3 may be localto or remote from each other. The components illustrated in FIG. 3 maybe implemented in software and/or hardware. Each component may bedistributed over multiple applications and/or machines. Multiplecomponents may be combined into one application and/or machine.Operations described with respect to one component may instead beperformed by another component.

In one or more embodiments, user information 330 includes informationrelated to a particular user. User information 330 may be collected fromone or more data sources. Data sources may include cookies associatedwith a user, web applications with personal information entered and/orshared by a user, and public data sources with public informationassociated with a user. User information 330 may specify a confirmableuser attribute. Additionally or alternatively, user information 330 maybe used to infer an inferred user attribute. Additionally oralternatively, user information 330 may be used to determine an affinityattribute.

In one or more embodiments, an inferred user attribute model 332 refersto hardware and/or software configured to determine one or more inferreduser attribute scores 336 for a user based on user information 330 ofthe user. An inferred user attribute model 332 may include a single datamodel for determining one or more inferred user attribute scores 336.Alternatively, an inferred user attribute model 332 may include multipledata models for determining respective inferred user attribute scores336. An inferred user attribute model 332 is implemented on one or moredigital devices.

An inferred user attribute score 336 may be expressed within any type ofdata range. As an example, an inferred user attribute score may be aninteger within the range 0 to 100. A higher score may represent a higherconfidence level in the determination of the inferred user attribute.Additionally or alternatively, a higher score may represent a higherdegree to which a user exhibits the inferred user attribute (forexample, a user is an extreme sports lover, not simply a moderate sportslover). As another example, an inferred user attribute score may be abinary value, such as 0 or 1, or true or false.

In an embodiment, an inferred user attribute model 332 is trained,generated, and/or updated using machine learning. A training set fortraining an inferred user attribute model 332 includes historical userinformation of various users collected from one or more data sources.Each set of historical user information, associated with a respectiveuser, is labeled with one or more inferred user attributes of the user.The labeled inferred user attributes may be determined via user inputand/or another application.

As an example, user information of a user may indicate that the userlives at 123 Main Street, Springfield; the user recently bought aminivan; the user is on a school board of a public elementary school.The user information may be labeled with inferred user attributes, suchas, the user is a parent; the user has more than one child; at least onechild is in elementary school.

In one or more embodiments, a machine learning algorithm 334 is analgorithm that can be iterated to learn a target model f that best mapsa set of input variables to an output variable. In particular, a machinelearning algorithm 334 is configured to generate and/or train aninferred user attribute model 332. A machine learning algorithm 334generates an inferred user attribute model 332 such that the inferreduser attribute model 332 best fits the historical user information tothe labeled inferred user attributes. Additionally or alternatively, amachine learning algorithm 334 generates an inferred user attributemodel 332 such that when the inferred user attribute model 332 isapplied to the historical user information, a maximum number of resultsdetermined by the inferred user attribute model 332 matches the labeledinferred user attributes.

A machine learning algorithm 334 may include supervised componentsand/or unsupervised components. Various types of algorithms may be used,such as linear regression, logistic regression, linear discriminantanalysis, classification and regression trees, naïve Bayes, k-nearestneighbors, learning vector quantization, support vector machine,decision trees, bagging and random forest, boosting, backpropagation.Examples of operations for applying various machine learning algorithms334 are further described below with reference to FIG. 7.

FIG. 4 illustrates a machine learning system for generating an affinityattribute model, in accordance with one or more embodiments. Asillustrated in FIG. 4, a system includes user information 430, anaffinity attribute model 432, a machine learning algorithm 434, and oneor more inferred user attributes 438. In one or more embodiments, thesystem may include more or fewer components than the componentsillustrated in FIG. 4. The components illustrated in FIG. 4 may be localto or remote from each other. The components illustrated in FIG. 4 maybe implemented in software and/or hardware. Each component may bedistributed over multiple applications and/or machines. Multiplecomponents may be combined into one application and/or machine.Operations described with respect to one component may instead beperformed by another component.

In one or more embodiments, an affinity attribute model 432 refers tohardware and/or software configured to determine one or more affinityattribute scores 438 for a user based on user information 430 of theuser. An affinity attribute model 432 may include a single data modelfor determining one or more affinity attribute scores 438 for one ormore product attributes. A single data model may determine positiveaffinity scores and/or negative affinity scores. As an example, a singleaffinity attribute model may determine a positive affinity attributescore for floral patterns and a positive affinity score for the colorblack. Alternatively, an affinity attribute model 432 may includemultiple data models for determining respective affinity attributescores 438 for respective product attributes. As an example, oneaffinity attribute model may determine a positive affinity towards sedancars, another affinity attribute model may determine a negative affinitytowards sedan cars, another affinity attribute model may determine apositive affinity towards hatchback cars, and another affinity attributemodel may determine a negative affinity towards hatchback cars. Aninferred user attribute model 332 is implemented on one or more digitaldevices.

An affinity attribute score 438 may be expressed within any type of datarange. As an example, an affinity attribute score may be an integerwithin the range 0 to 100. A higher score may represent a higherconfidence level in the determination of the affinity attribute.Additionally or alternatively, a higher score may represent a higherdegree to which a user exhibits the affinity attribute (for example, auser has an extreme affinity towards floral patterns, not simply amoderate affinity towards floral patterns). As another example, anaffinity attribute score may be a binary value, such as 0 or 1, or trueor false.

In an embodiment, an affinity attribute model 432 is trained, generated,and/or updated using machine learning. A training set for training anaffinity attribute model 432 includes historical user information ofvarious users collected from one or more data sources. The historicaluser information may also include inferred user attributes. Each set ofhistorical user information, associated with a respective user, islabeled with one or more affinity attributes of the user. The labeledaffinity attributes may be determined via user input and/or anotherapplication.

As an example, user information of a user may indicate that the userlives at 123 Main Street, Springfield; the user made recent purchases oforganic hair products; and the user made recent purchases of organicvegetables. The user information may be labeled with affinityattributes, such as, the user has a positive affinity towards organicproducts, and the user has a negative affinity towards non-organicproducts.

In one or more embodiments, a machine learning algorithm 434 is analgorithm that can be iterated to learn a target model f that best mapsa set of input variables to an output variable. In particular, a machinelearning algorithm 434 is configured to generate and/or train anaffinity attribute model 432. A machine learning algorithm 434 generatesan affinity attribute model 432 such that the affinity attribute model432 best fits the historical user information to the labeled affinityattributes. Additionally or alternatively, a machine learning algorithm434 generates an affinity attribute model 432 such that when theaffinity attribute model 432 is applied to the historical userinformation, a maximum number of results determined by the affinityattribute model 432 matches the labeled affinity attributes.

A machine learning algorithm 434 may include supervised componentsand/or unsupervised components. Various types of algorithms may be used,such as linear regression, logistic regression, linear discriminantanalysis, classification and regression trees, naïve Bayes, k-nearestneighbors, learning vector quantization, support vector machine,decision trees, bagging and random forest, boosting, backpropagation.Examples of operations for applying various machine learning algorithms434 are further described below with reference to FIG. 7.

In an embodiment, the inferred user attribute model 332 and the affinityattribute model 432 include different data models for determininginferred user attributes and affinity attributes, respectively. Inanother embodiment, the inferred user attribute model 332 and theaffinity attribute model 432 include the same data models fordetermining inferred user attributes and affinity attributes.

In an embodiment, the same machine learning algorithm is used to trainthe inferred user attribute model 332 and the affinity attribute model432. In another embodiment, different machine learning algorithms areused to train the inferred user attribute model 332 and the affinityattribute model 432.

4. Determining A Target Group of Users for Engaging with A Campaign

FIG. 5 illustrates an example set of operations for determining a targetgroup of users for engaging with a campaign, in accordance with one ormore embodiments. One or more operations illustrated in FIG. 5 may bemodified, rearranged, or omitted all together. Accordingly, theparticular sequence of operations illustrated in FIG. 5 should not beconstrued as limiting the scope of one or more embodiments.

One or more embodiments include obtaining a campaign profile specifyingproducts, content items, and/or campaign attributes associated with acampaign (Operation 502). A target group selection system obtains acampaign profile from a data repository. The campaign profile specifiesproducts that are being promoted and/or discounted in the campaign.Additionally or alternatively, the campaign profile specifies contentitems that would be transmitted to promote and/or bring attention to theproducts. Additionally or alternatively, the campaign profile specifiescampaign attributes of the campaign.

One or more embodiments include obtaining product attributes of theproducts and/or content attributes of the content items (Operation 504).The target group selection system obtains product profiles of theproducts from a data repository. A product profile of a particularproduct specifies products attributes of the particular product.Additionally or alternatively, the target group selection system obtainsproduct attributes that are specified in the campaign profile itself.Additionally or alternatively, the target group selection systemanalyzes the products to determine product attributes. The target groupselection system may, for example, analyze labels of a particularproduct, an outward appearance of the particular product, and/or performtesting on the particular product. Based on the analysis, the targetgroup selection system determines product attributes.

The target group selection system obtains content attributes that arespecified in the campaign profile itself. As an example, a campaignprofile may specify that the campaign is a clearance campaign. A targetgroup selection system may thus determine “clearance” as a contentattribute. Additionally or alternatively, the target group selectionsystem analyzes the content items to determine content attributes. Thetarget group selection system may, for example, perform natural languageprocessing, keyword search, and/or other analysis on the content items.The target group selection system may, for example, analyze colors,logos, graphics, and/or fonts included in content items. Based on theanalysis, the target group selection system determines contentattributes.

One or more embodiments include presenting, on a user interface, theproduct attributes, content attributes, and/or campaign attributes(Operation 506). The target group selection system presents the productattributes and/or content attributes on a user interface. An example ofa user interface is described below with reference to FIGS. 8A-B.

One or more embodiments include obtaining, via the user interface, userinput specifying target user tuning parameters (Operation 508). Thetarget group selection system obtains user input specifying target usertuning parameters (such as an inclusion range, inclusion weight,exclusion range, and/or exclusion weight) via the user interface. Thetarget group selection system obtains user input specifying respectivetarget user tuning parameters for confirmable user attributes, inferreduser attributes, and/or affinity attributes. Target user tuningparameters may but are not necessarily specified for every userattribute included in a user profile.

In an embodiment, the target group selection system obtains target usertuning parameters for product-specific affinity attributes,content-specific affinity attributes, and/or campaign-specific affinityattributes. As described above, product attributes, content attributes,and/or campaign attributes associated with the campaign are determinedat Operations 502-504. The target group selection system may receiveinclusion ranges and/or inclusion weights for positive affinityattributes towards one or more of the product attributes, contentattributes, and/or campaign attributes associated with the campaign. Thetarget group selection system may receive exclusion ranges and/orexclusion weights for negative affinity attributes towards one or moreof the product attributes, content attributes, and/or campaignattributes associated with the campaign. The target group selectionsystem may receive inclusion ranges and/or inclusion weights fornegative affinity attributes towards one or more product attributes,content attributes, and/or campaign attributes that are not associatedwith the campaign. The target group selection system may receiveexclusion ranges and/or exclusion weights for positive affinityattributes towards one or more product attributes, content attributes,and/or campaign attributes that are not associated with the campaign.

As an example, a campaign may promote products with striped patterns. Anadministrator may determine that users who favor striped patterns shouldbe included in a target group. The administrator may hence specify aninclusion weight applicable to a positive affinity attribute towardsstriped patterns. The administrator may determine that users whodisfavor striped patterns should be excluded from the target group. Theadministrator may hence specify an exclusion weight applicable to anegative affinity attribute towards striped patterns. Hence, a targetgroup selection system obtains target user tuning parameters forproduct-specific affinity attributes, content-specific affinityattributes, and/or campaign-specific affinity attributes via a userinterface.

As another example, a product attribute hierarchy may include a top nodeassociated with a product type, “clothing.” Two child nodes of the“clothing” node may be respectively associated with the product types,“tops” and “bottoms.” Two child nodes of the “bottoms” node may berespectively associated with the styles, “solid colors” and “patterned.”Two child nodes of the “patterned” node may be respectively associatedwith the patterns, “striped pattern” and “polka dot pattern.”

A campaign may promote products with striped patterns. Based on theproduct attribute hierarchy, product attributes associated with thecampaign include: “striped pattern,” “patterned,” “bottoms,” and“clothing.” Product attributes not associated with the campaign include,for example: “polka dot pattern,” “solid colors,” and “tops.” Otherproduct attributes not listed in the product attribute hierarchy mayalso be considered as being not associated with the campaign, such as“floral pattern,” and “shoes.”

An administrator may determine that users who favor the “patterned”style should be included in a target group. The administrator may hencespecify an inclusion weight applicable to a positive affinity attributetowards “patterned.”

The administrator may determine that users who disfavor the “stripedpattern” should be excluded from the target group. The administrator mayhence specify an exclusion weight applicable to a negative affinityattribute towards “striped pattern.”

The administrator may determine that users who favor the “solid colors”style should be excluded from the target group. The administrator mayhence specify an exclusion weight applicable to a positive affinitytowards “solid colors.”

The administrator may determine that users who disfavor “tops” should beincluded in the target group. The administrator may hence specify aninclusion weight applicable to a negative affinity towards “tops.”

As shown in the above example, a target group selection system obtainstarget user tuning parameters for product-specific affinity attributes,content-specific affinity attributes, and/or campaign-specific affinityattributes via a user interface.

Hence, target user tuning parameters may specify a degree to whichaffinity towards product attributes specific to the campaign are takeninto account when determining a target group of users. Additionally oralternatively, the target user tuning parameters may specify a degree towhich affinity towards content attributes specific to the campaign aretaken into account when determining the target group of users.Additionally or alternatively, the target user tuning parameters mayspecify a degree to which affinity towards campaign attributes specificto the campaign are taken into account when determining the target groupof users.

In an embodiment, the user interface for receiving target user tuningparameters is a graphical user interface (GUI) configured to acceptdrag-and-drop user input. The GUI may include a selection region and aconfiguration region. The target group selection system may present theproduct attributes, content attributes, and/or campaign attributes inthe selection region. An administrator may select an icon representing aparticular product attribute, content attribute, and/or campaignattribute from the selection region and drag the icon to theconfiguration region. The administrator may then enter target usertuning parameters applicable to affinity attributes for the particularproduct attribute and/or content attribute. Additionally oralternatively, the administrator may enter user attributes, notdisplayed in the selection region, into the configuration region. Theadministrator may review the displayed product attributes and/or contentattributes to determine confirmable user attributes and/or inferred userattributes relevant to the campaign. The administrator may then entertarget user tuning parameters applicable to the relevant userattributes. An example of a user interface is described below withreference to FIGS. 8A-B.

In an embodiment, the target group selection system generates aninclusion score function and/or exclusion score function based on thetarget user tuning parameters. The inclusion score function and/orexclusion score function specifies which target user tuning parametersare applicable to which user attributes of a user profile.

In an alternative embodiment, the target group selection systemdetermines target user tuning parameters based on a campaign profile,without a user directly specifying the target user tuning parameters viaa user interface. In particular, the target group selection systemobtains a campaign profile (which may have been specified via user inputand/or by another application). The target group selection systemdetermines products associated with the campaign based on the campaignprofile, without further user input. The target group selection systemdetermines product attributes of the products based on product profilesand/or product analysis, without user input. The target group selectionsystem determines content items associated with the campaign based onthe campaign profile, without further user input. The target groupselection system determines content attributes of the content itemsbased on content analysis, without user input. The target groupselection system determines campaign attributes based on the campaignprofile, without further user input. The target group selection systemthen determines inclusion weights for affinity attributes towards theproduct attributes, content attributes, and/or campaign attributes,based on various possible methods, examples of which are describedbelow.

The inclusion weights for affinity attributes towards the productattributes, content attributes, and/or campaign attributes may be equal.For example, if a campaign is associated with two product attributes andthree content attributes, inclusion weights for affinity towards eachattribute may be 20%.

Additionally or alternatively, the inclusion weights for affinityattributes towards the product attributes may be determined based on aproduct attribute hierarchy. Greater weight may be given to productattributes that are more specific, that is, product attributes that arelower in the product attribute hierarchy. As an example, a datarepository may store two product attribute hierarchies. A first productattribute hierarchy may include a top node associated with “clothing.”Two child nodes of the “clothing” node may be respectively associatedwith “tops” and “bottoms.” A second product attribute hierarchy mayinclude a top node associated with “patterned.” Two child nodes of the“patterned” node may be respectively associated with “striped pattern”and “polka dot pattern.”

A product profile of a product associated with a campaign may specifythe following product attributes: “bottoms” and “patterned.” The“bottoms” attribute is associated with a second level in the firsthierarchy, while the “patterned” attribute is associated with a firstlevel in the second hierarchy. Therefore, the “bottoms” attribute isassociated with a lower hierarchical level. Hence, a greater inclusionweight may be given to “bottoms” than to “patterned.” The difference ininclusion weights to “bottoms” and “patterned” may be proportional tothe difference in hierarchical levels associated with “bottoms” and“patterned.” Hence, inclusion weights are determined for affinityattributes towards “bottoms” and “patterned,” without user input.

Additionally or alternatively, the inclusion weights for affinityattributes towards the product attributes may be determined based onrevenue and/or profit generated by products associated with the productattributes. As an example, a sales application may indicate that therevenue generated by each of a list of products (such as bicycles).Product profiles may specify product attributes of each bicycle. Hence,a target group selection system may determine that bicycles with theproduct attribute “with kickstands” generate a greater revenue thanbicycles without the product attribute. Meanwhile, the target groupselection system may determine that there is an insignificant differencein generated revenue for bicycles of different colors. Hence, the targetgroup selection system may apply a greater inclusion weight for anaffinity attribute towards “with kickstands,” as compared to inclusionweights for an affinity attribute towards “blue,” “pink,” and othercolors.

One or more embodiments include determining an inclusion group based onthe target user tuning parameters (Operation 510). The target groupselection system determines an inclusion group of users by applying thetarget user tuning parameters to user profiles of a set of users.Examples of operations for determining an inclusion group are describedbelow with reference to FIG. 6A.

One or more embodiments include determining an exclusion group based onthe target user tuning parameters (Operation 512). The target groupselection system determines an exclusion group of users by applying thetarget user tuning parameters to the user profiles of the set of users.Examples of operations for determining an inclusion group are describedbelow with reference to FIG. 6B.

One or more embodiments include determining, for each user in theinclusion group, whether the user is also in the exclusion group(Operation 514). The target group selection system traverses througheach user in the inclusion group to determine whether the user is alsoin the exclusion group.

If a particular user is in the inclusion group and in the exclusiongroup, one or more embodiments include excluding the particular userfrom the target group (Operation 520). The target group selection systemexcludes each user that is in the inclusion group and in the exclusiongroup from the target group of users.

The target group selection system does not add any user who is not inthe inclusion group into the target group of users.

If a particular user is in the inclusion group but not in the exclusiongroup, one or more embodiments include adding the particular user intothe target group (Operation 516). The target group selection system addseach user that is in the inclusion group but not in the exclusion groupinto the target group of users.

One or more embodiments include transmitting the content items to thetarget group (Operation 518). The target group selection systemtransmits the content items to the target group of users. The targetgroup selection system does not transmit the content items to otherusers who are not in the target group.

The target group selection system may, for example, transmit emailsincluding the content items to the target group. The target groupselection system may present video content and/or audio content to thetarget group. The target group selection system may cause mailing labelsto be printed for the target group, for physically mailing the contentitems to the target group.

Based on the operations of FIG. 5, the target group selection systemdetermines a target group of users for engaging with a campaign. In anembodiment, the target group is selected based at least onproduct-specific affinity attributes and/or content-specific affinityattributes. In an embodiment, the target group is selected based on aweighted consideration of product-specific affinity attributes,content-specific affinity attributes, and/or other user attributes.

FIGS. 6A-B illustrate example sets of operations for determining aninclusion group of users and an exclusion group of users, in accordancewith one or more embodiments. One or more operations illustrated in FIG.6A-B may be modified, rearranged, or omitted all together. Accordingly,the particular sequence of operations illustrated in FIGS. 6A-B shouldnot be construed as limiting the scope of one or more embodiments.

Referring to FIG. 6A, one or more embodiments include obtaining a userprofile, of a user, specifying confirmable user attributes, inferreduser attributes, and/or affinity attributes (Operation 602). A targetgroup selection system obtains a user profile of a user.

The target group selection system may retrieve the user profile from adata repository. The user profile may be generated based on user inputand/or another application.

Additionally or alternatively, the target group selection system maygenerate and/or enhance the user profile. Examples of operations forusing machine learning to generate and/or enhance a user profile aredescribed below with reference to FIG. 7. Additional and/or alternativemethods for generating and/or enhancing a user profile may be used. Asan example, a rule-based system, a lookup table, and/or a user-definedfunction may be used for generating a user profile.

One or more embodiments include identifying attributes to be evaluatedusing inclusion ranges (Operation 604). The target group selectionsystem identifies attributes to be evaluated using inclusion ranges, asspecified by a set of target user tuning parameters. If the target usertuning parameters specify an inclusion range for a particular attribute(such as a confirmable user attribute, inferred user attribute, and/oraffinity attribute), then the particular attribute is evaluated usingthe inclusion range. If the target user tuning parameters do not specifyan inclusion range for a particular attribute (such as a confirmableuser attribute, inferred user attribute, and/or affinity attribute),then the particular attribute is not evaluated using any inclusionrange.

One or more embodiments include applying inclusion ranges to theattributes (Operation 606). The target group selection system appliesthe inclusion ranges respectively to the attributes.

As an example, a set of target user tuning parameters may specify thefollowing:

TABLE 1 User Attribute Inclusion Range/Weight Value The user is acollege student Weight 50% The user has a positive Range 90-100 affinitytowards BrandXYZ computer products

Based on the target user tuning parameters, a target group selectionsystem may evaluate the user attribute, “college student,” using aninclusion weight. The target group selection system may evaluate theuser attribute, “positive affinity towards BrandXYZ computer products,”using an inclusion range. The target group selection system may applythe range [90-100] to the user attribute, “positive affinity towardsBrandXYZ computer products.”

In an embodiment, the target group selection system obtains an inclusionscore function, generated based on one or more target user tuningparameters. The inclusion score function is configured to apply one ormore inclusion ranges respectively to one or more user attributes. Thetarget group selection system inputs user attributes specified in theuser profile into the inclusion score function. Based on the inclusionscore function, the target group selection system applies inclusionranges to the applicable user attributes.

One or more embodiments include determining whether the attributes matchan inclusion scope (Operation 608). The target user tuning parametersmay specify an inclusion scope that requires all inclusion ranges to besatisfied. Alternatively, the target user tuning parameters may specifyan inclusion scope that requires only a subset of inclusion ranges to besatisfied. If the inclusion scope is satisfied, then the user is addedinto the inclusion group (Operation 618). The target group selectionsystem may optionally determine an “inclusion score” for the user thatis above the threshold value evaluated at Operation 616 described below.If the inclusion scope is not satisfied, then the user is excluded fromthe inclusion group (Operation 620). The target group selection systemmay optionally determine an “inclusion score” for the user that is notabove the threshold value evaluated at Operation 616 described below.

As an example, target user tuning parameters may include applying arange [90-100] to a user attribute, “positive affinity towards BrandXYZcomputer products” and applying a range [85-100] to a user attribute,“negative affinity towards cartoon characters.”

If an inclusion scope requires all inclusion ranges to be satisfied,then a user is included in an inclusion group only if the user has both(a) a 90-100 positive affinity score towards BrandXYZ computer products,and (b) an 85-100 negative affinity score towards cartoon characters.

If an inclusion scope requires only one inclusion range to be satisfied,then a user is included in an inclusion group if the user has either (a)a 90-100 positive affinity score towards BrandXYZ computer products, or(b) an 85-100 negative affinity score towards cartoon characters.

As another example, target user tuning parameters may include applying arange [90-100] to a user attribute, “positive affinity towards BrandXYZcomputer products”; applying a range [95-100] to a user attribute,“positive affinity towards high-end audio products”; and applying arange [85-100] to a user attribute, “negative affinity towards cartooncharacters.” The target user tuning parameters may further specify aninclusion scope that requires:

(a) satisfying the range [90-100] for the user attribute, “positiveaffinity towards BrandXYZ computer products”; or(b) satisfying the range [95-100] for the user attribute, “positiveaffinity towards high-end audio products,” and the range [85-100] forthe user attribute, “negative affinity towards cartoon characters.”

A user Joe may have a positive affinity score of 95 towards BrandXYZcomputer products, a positive affinity score of 70 towards high-endaudio products, and an 89 negative affinity score towards cartooncharacters. Since Joe satisfies the range [90-100] for the userattribute, “positive affinity towards BrandXYZ computer products,” atarget group selection system adds Joe to an inclusion group.

Another user Mary may have a positive affinity score of 86 towardsBrandXYZ computer products, a positive affinity score of 96 towardshigh-end audio products, and a 70 negative affinity score towardscartoon characters. Mary does not satisfy the range [90-100] for theuser attribute, “positive affinity towards BrandXYZ computer products.”Further, although Mary satisfies the range [95-100] for the userattribute, “positive affinity towards high-end audio products,” Marydoes not satisfy the range [85-100] for the user attribute, “negativeaffinity towards cartoon characters.” Therefore, the target groupselection system does not add Mary to the inclusion group.

Another user Janet may have a positive affinity score of 84 towardsBrandXYZ computer products, a positive affinity score of 97 towardshigh-end audio products, and a 90 negative affinity score towardscartoon characters. Since Janet satisfies the range [95-100] for theuser attribute, “positive affinity towards high-end audio products,” andthe range [85-100] for the user attribute, “negative affinity towardscartoon characters,” the target group selection system adds Janet to theinclusion group.

One or more embodiments include identifying attributes to be evaluatedusing inclusion weights (Operation 610). The target group selectionsystem identifies attributes to be evaluated using inclusion weights, asspecified by the set of target user tuning parameters. If the targetuser tuning parameters specify an inclusion weight for a particularattribute (such as a confirmable user attribute, inferred userattribute, and/or affinity attribute), then the particular attribute isevaluated using the inclusion weight. If the target user tuningparameters do not specify an inclusion weight for a particular attribute(such as a confirmable user attribute, inferred user attribute, and/oraffinity attribute), then the particular attribute is not evaluatedusing any inclusion weight. A particular attribute may be evaluated byan inclusion range; an inclusion weight; both an inclusion range and aninclusion weight; or neither an inclusion range nor an inclusion weight.

As an example, a set of target user tuning parameters may specify thefollowing:

TABLE 2 User Attribute Inclusion Range/Weight Value The user is acollege student Weight 60% The user has a positive Weight 40% affinitytowards BrandXYZ computer products The user has a positive Range 90-100affinity towards BrandXYZ computer products

A user profile for John may indicate an inferred user attribute scorefor “college student” is 65. The user profile may indicate a positiveaffinity score towards BrandXYZ computer products is 55.

A target group selection system may first apply the inclusion range[90-100] to the user attribute, “positive affinity towards BrandXYZcomputer products.” Since John's positive affinity score for BrandXYZcomputer products is 55, the inclusion range is not satisfied.

The target group selection system may then apply inclusion weights. Thetarget group selection system may apply the weight 60% to the userattribute, “college student,” and the weight 40% to the user attribute,“positive affinity towards BrandXYZ computer products.” The target groupselection system may compute the sum 0.6×65+0.4×55=61. The inclusionscore for John may hence be determined as 61.

In the above example, the user attribute, “positive affinity towardsBrandXYZ computer products,” is evaluated first with respect to aninclusion range and then with respect to an inclusion weight. Hence, ifa user attribute does not satisfy an inclusion range, the user attributemay still be taken into account when computing an inclusion score.

One or more embodiments include applying inclusion weights to theattributes (Operation 612). The target group selection system appliesthe inclusion weights respectively to the attributes.

As an example, a set of target user tuning parameters may specify thefollowing:

TABLE 3 User Attribute Inclusion Range/Weight Value The user is acollege student Weight 50% The user has a positive Range 90-100 affinitytowards BrandXYZ computer products

Based on the target user tuning parameters, a target group selectionsystem may evaluate the user attribute, “college student,” using aninclusion weight. The target group selection system may evaluate theuser attribute, “positive affinity towards BrandXYZ computer products,”using an inclusion range. The target group selection system may applythe weight 50% to the user attribute, “college student.”

In an embodiment, the target group selection system obtains an inclusionscore function, generated based on one or more target user tuningparameters. The inclusion score function is configured to apply one ormore inclusion weights respectively to one or more user attributes. Thetarget group selection system inputs user attributes specified in theuser profile into the inclusion score function. Based on the inclusionscore function, the target group selection system applies inclusionweights to the applicable user attributes.

One or more embodiments include determining an inclusion score for theuser (Operation 614). The target group selection system determines aninclusion score for the user. In an embodiment, the inclusion score is aweighted sum of the attributes. As an example, an inclusion weight of60% may be applicable to the user attribute, “college student.” Aninclusion weight of 40% may be applicable to the user attribute,“positive affinity towards BrandXYZ computer products.” A target groupselection system may compute the weighted sum, 0.6×[inferred userattribute score for “college student”]+0.4×[positive affinity score forBrandXYZ computer products], to determine an inclusion score. Additionaland/or alternative functions for determining an inclusion score based onapplying respective weights to respective attributes may be used.

In an embodiment, by applying weights to user attributes, a user is notnecessarily excluded from an inclusion group based on a hard filter. Asan example, an inclusion score function may be 0.4×[inferred userattribute score for “college student”]+0.2×[confirmable user attributefor “female”]+0.1×[confirmable user attribute for “age between 18 and25”]+0.1×[positive affinity towards “floral patterns”]+0.2×[negativeaffinity towards “short dresses”]. Based on the above inclusion scorefunction, the “college student” inferred user attribute is considered asignificant factor for including a user in an inclusion group. However,even a user who is not a college student may be included in theinclusion group, if the user is a female, is between 18 and 25 yearsold, has positive affinity towards “floral patterns,” and/or hasnegative affinity towards “short dresses.”

One or more embodiments include determining whether the inclusion scoreis above a threshold value (Operation 616). The target group selectionsystem compares the inclusion score with a threshold value. If theinclusion score is above the threshold value, then the user is addedinto the inclusion group (Operation 618). If the inclusion score is notabove the threshold value, then the user is excluded from the inclusiongroup (Operation 620).

Referring to FIG. 6B, the operations of FIG. 6B are similar to theoperations of FIG. 6A. The operations of FIG. 6B are applied withrespect to exclusion ranges, exclusion weights, an exclusion score, andan exclusion group, whereas the operations of FIG. 6A are applied withrespect to inclusion ranges, inclusion weights, an inclusion score, andan inclusion group

One or more embodiments include obtaining a user profile, of a user,specifying confirmable user attributes, inferred user attributes, and/oraffinity attributes (Operation 622). Examples of operations forobtaining a user profile are described above with reference to Operation602.

One or more embodiments include identifying attributes to be evaluatedusing exclusion ranges (Operation 624). Examples of operations foridentifying attributes to be evaluated using ranges (such as inclusionranges) are described above with reference to Operation 604.

One or more embodiments include applying exclusion ranges to theattributes (Operation 626). Examples of operations for applying ranges(such as inclusion ranges) to attributes are described above withreference to Operation 606.

One or more embodiments include determining whether the attributes matchan exclusion scope (Operation 628). Examples of operations fordetermining whether attributes match a scope (such as an inclusionscope) are described above with reference to Operation 608.

If the exclusion scope is satisfied, then the user is added into theexclusion group (Operation 638). The target group selection system mayoptionally determine an “exclusion score” for the user that is above thethreshold value evaluated at Operation 636 described below. If theexclusion scope is not satisfied, then the user is excluded from theexclusion group (Operation 620). The target group selection system mayoptionally determine an “exclusion score” for the user that is not abovethe threshold value evaluated at Operation 636 described below.

One or more embodiments include identifying attributes to be evaluatedusing exclusion weights (Operation 630). Examples of operations foridentifying attributes to be evaluated using weights (such as exclusionweights) are described above with reference to Operation 610.

One or more embodiments include applying inclusion weights to theattributes (Operation 632). Examples of operations for applying weights(such as inclusion weights) to attributes are described above withreference to Operation 612.

One or more embodiments include determining an inclusion score for theuser (Operation 634). Examples of operations for determining a score(such as an inclusion score) for a user are described above withreference to Operation 614.

One or more embodiments include determining whether the exclusion scoreis above a threshold value (Operation 636). The target group selectionsystem compares the exclusion score with a threshold value. If theexclusion score is above the threshold value, then the user is addedinto the exclusion group (Operation 638). If the exclusion score is notabove the threshold value, then the user is excluded from the exclusiongroup (Operation 640).

In an embodiment, the target group selection engine may iterate theoperations of FIG. 6A and/or FIG. 6B with respect to each user profile.The target group selection engine may hence determine whether to addeach user, associated with each user profile, to the inclusion groupand/or exclusion group.

5. Using Machine Learning to Generate Models for Generating and/orEnhancing User Profiles

FIG. 7 illustrates an example set of operations for using machinelearning to generate models for generating and/or enhancing userprofiles, in accordance with one or more embodiments. One or moreoperations illustrated in FIG. 7 may be modified, rearranged, or omittedall together. Accordingly, the particular sequence of operationsillustrated in FIG. 7 should not be construed as limiting the scope ofone or more embodiments.

One or more embodiments include obtaining user information for aparticular user (Operation 702). A target group selection system obtainsuser information from one or more data sources. Data sources may includecookies associated with the particular user, web applications withpersonal information entered and/or shared by the particular user,public data sources with public information associated with theparticular user, and/or any other sources.

One or more embodiments include determining confirmable user attributesbased on the user information (Operation 704). The target groupselection system determines confirmable user attributes based on theuser information. As an example, Alan may share personal information ona social media website, indicating that her birthday is Dec. 12, 1982. Atarget group selection system may obtain the personal information fromthe social media website. The target group selection system maydetermine that a confirmable user attribute of Alan is that Alan'sbirthday is on Dec. 12, 1982.

In an embodiment, the target group selection system checks whichconfirmable user attributes are required as input for determiningwhether to add the user to the inclusion group and/or exclusion group,as described above with reference to FIGS. 6A-B. In particular, thetarget group selection system checks which attributes are to beevaluated using inclusion ranges and/or inclusion weights at Operations604 and 610. The target group selection system checks which attributesare to be evaluated using exclusion ranges and/or exclusion weights atOperations 624 and 630. The target group selection system then analyzesthe user information obtained at Operation 702 to determine the requiredconfirmable user attributes. The target group selection system may butdoes not necessarily determine any other confirmable user attributesfrom the user information.

One or more embodiments include generating an inferred user attributemodel via machine learning (Operation 706). A machine learning system(which may be but is not necessarily a part of the target groupselection system) generates an inferred user attribute model via machinelearning.

The machine learning system obtains a training set for training theinferred user attribute model. The machine learning system obtainshistorical user information from one or more data sources. As describedabove, data sources may include cookies associated with the particularuser, web applications with personal information entered and/or sharedby the particular user, public data sources with public informationassociated with the particular user, and/or any other sources. Thetraining set is labeled with inferred user attributes, which may bedetermined based on user input and/or another application.

As an example, a machine learning system may obtain the followingtraining set:

TABLE 4 Labeled Inferred User User Information Attribute Name: MarySports Lover Birthday: Dec. 6, 1982 Investor Occupation: Attorney WebInteraction History: Visited sportsnews.com on Oct. 1, 2019; visitedinvestment.com on Oct. 2, 2019; visited basketballteam.com on Oct. 2,2019 Name: John Sports Lover Geographical Location: Los Angeles, CAFoodie Education: Graduated UCLA in 2007 Web Interaction History:Visited uclafootballteam.com on Oct. 3, 2019; visited restaurantabc.comon Oct. 3, 2019 Name: [Not Found] Extrovert Geographical Location:Burbank, CA Birthday: May 5, 2001 Web Interaction History: Visitedsocialmediabook.com on Oct. 2, 2019; visited socialmediabook.com on Oct.3, 2019

In an embodiment, the labeled inferred user attributes are alsoassociated with a score. The score may indicate a degree to which aparticular set of user information exhibits the labeled inferred userattribute. In the example training set above, for example, the first setof user information (associated with Mary) may be associated with thelabeled inferred user attribute “Sports Lover” and a score of 80. Thesecond set of user information (associated with John) may be associatedwith the labeled inferred user attribute “Sports Lover” and a score of70. The scores indicate that Mary is more of a sports lover than John.

The machine learning system applies a machine learning algorithm to thetraining set to generate an inferred user attribute model. Variousmachine learning algorithms may be used, examples of which are describedbelow.

In an embodiment, a regression algorithm is used. The machine learningsystem determines a regression function form. The regression functionform includes a set of weights applicable to a set of raw attributesderived from a set of user information. As an example, each geographicallocation may be coded as a unique numerical value. Los Angeles may becoded as 1 and Burbank may be coded as 2. The encoded geographicallocations may be used as attributes in a regression function form. Asanother example, each geographical location may be coded based on adistance from a reference location (such as a store address). Theencoded geographical locations may be used as attributes in a regressionfunction form.

As an example, a regression function form may be:

Y=β ₀+β₁ x ₁+β₂ x ₂+ . . . +ε

wherein Y is the output variable, which in this case, are the inferreduser attributes; x_(i) are the attributes derived from user information;c is an error variable; and β_(i), are the weights applied to theattributes.

Alternative regression function forms may also be used. Variousregression function forms may be associated with different types ofregression, such as ordinary least squares regression, linearregression, non-linear regression, logistic regression, stepwiseregression, polynomial regression, binomial regression, binaryregression, non-parametric regression, multivariate adaptive regressionspline, and locally estimated scatterplot smoothing regression.

Additionally, the machine learning system determines an errormeasurement function. The error measurement function is configured todetermine an error between (a) a set of inferred user attributesdetermined by applying the regression function form, having a particularset of values for the set of weights, to the historical userinformation, and (b) the labeled inferred user attributes. An error maybe represented as:

e=Y−Ŷ

wherein Y represents values of the output variable from a regressionfunction based on the historical user information; and Ŷ represents thevalues of the labeled inferred user attributes.

As an example, an error estimation function may be:

${SSR} = {\sum\limits_{i = 1}^{n}e_{i}^{2}}$

Alternative error measurement functions may also be used. An example ofan error measurement function is a sum of squared differences between(a) the set of inferred user attributes determined by applying aregression function to the historical user information, and (b) thelabeled user attributes.

Thereafter, the machine learning system determines a “best” set ofvalues for the set of weights β_(i) in the regression function form,such that the error, determined by the error measurement function, isminimized. The best values for the set of weights β_(i) are applied tothe regression function form to generate a regression function. Theinferred user attribute model may utilize the regression function todetermine inferred user attributes for any user.

In an embodiment, a k-nearest neighbors (knn) algorithm is used. Themachine learning system generates the inferred user attribute model bystoring (a) attributes derived from the historical user information and(b) corresponding labeled inferred user attributes. The machine learningsystem may store the data as a set of numerical values and/or a graph.The data forms a knn data model. The inferred user attribute model mayutilize the knn data model to determine inferred user attributes for anyuser.

In an embodiment, a decision tree learning algorithm is used. Themachine learning system determines a split quality function. The splitquality function is configured to determine a quality of a split in adecision tree. A quality of a split may depend on the homogeneity of theoutput variable (which are the labeled inferred user attributes) withinthe subsets created by the split.

Thereafter the machine learning system iteratively selects a respectiveattribute derived from the historical user information as a respectivevariable for a respective split of the decision tree. The machinelearning system may work from the top down, from the root node to theleaf nodes. The machine learning system selects each attribute for eachsplit, such that a quality of the split as determined by the splitquality function is maximized.

As an example, a set of attributes may be derived from a set ofhistorical user information. A machine learning system may select anattribute, from the set of attributes, as a variable for a first splitof the decision tree, such that a quality of the first split asdetermined by the split quality function is maximized. Then the machinelearning system may select another attribute, from the remaining set ofattributes, as a variable for a second split of the decision tree, suchthat a quality of the second split as determined by the split qualityfunction is maximized. Hence, the machine learning system may iteratethe above procedure with respect to each split in the decision tree.

The machine learning system thereby generates a decision tree. Theinferred user attribute model may utilize the decision tree to determineinferred user attributes for any user. In an embodiment, the inferreduser attribute model may include multiple decision trees, may be used asa random forest decision tree.

In an embodiment, an artificial neural network (ANN) and/or deeplearning algorithm is used. The machine learning system generates one ormore input layers, one or more hidden layers, and an output layer of theANN. The inferred user attribute model may utilize the ANN to determineinferred user attributes for any user.

Each hidden layer includes one or more neurons and/or processing nodes.A neuron is associated with (a) a weighted summation and (b) anactivation function. The weighted summation is associated with a set ofweights, respectively applicable to a set of inputs to the neuron. Wherean input is a vector, the applicable weight is also a vector. A dotproduct is computed between each input vector and the applicable weightvector. The sum of the dot products (plus optionally a weighted biasvalue) may be referred to as a “state.” The activation function takesthe state as an input and normalizes the result (generally, a valuebetween 0 and 1). The activation function may be, for example, a sigmoidfunction (such as a logistic function, or a hyperbolic tangentfunction), or any other function. Generally, the weights and activationfunction are learned using machine learning (such as backpropagation),however other methods for setting the weights and activation functionmay be used.

The machine learning system determines types of hidden layers to beused. Types of hidden layers may include a fully connected layer, aconvolutional layer, a max pooling layer, and/or other layers.

A fully connected layer includes a set of neurons that are fullyconnected with a previous input layer or hidden layer. Each neuron ofthe fully connected layer is connected to each neuron of the previouslayer.

A convolutional layer includes a set of neurons, each of which isconnected only with a subset of neurons from a previous input layer orhidden layer. In particular, each neuron is associated with a filter ofa particular window size. A neuron of the convolutional layer isconnected with neurons of a previous layer that are within the filter.The filters of a set of neurons of a convolutional layer span across theneurons of the previous layer. As an example, a previous layer mayinclude five neurons: Neuron 1, Neuron 2, Neuron 3, Neuron 4, Neuron 5.A convolutional layer may include neurons associated with a filter ofsize three. Hence a first neuron of the convolutional layer may beconnected to Neuron 1, Neuron 2, Neuron 3; a second neuron of theconvolutional layer may be connected to Neuron 2, Neuron 3, Neuron 4; athird neuron of the convolutional layer may be connected to Neuron 3,Neuron 4, Neuron 5. The first, second, and third neurons of theconvolutional layer thereby span across the neurons of the previouslayer. The first, second, and third neurons together produce a vector.The vector includes three elements: one element determined based onNeuron 1, Neuron 2, Neuron 3; another element determined based on Neuron2, Neuron 3, Neuron 4; another element determined based on Neuron 3,Neuron 4, Neuron 5.

A max pooling layer includes a set of neurons associated with weightsthat are not learned via machine learning but are determined based onthe values of the inputs. In particular, the input with the maximumvalue has a weight of one, while all other inputs have a weight of zero.

Additional and/or alternative types of machine learning algorithms mayalso be used.

In an embodiment, the inferred user attribute model includes a singledata model to determine one or more inferred user attributes. In anotherembodiment, the inferred user attribute model includes different datamodels to determine different inferred user attributes. The differentdata models may use the same type of machine learning algorithm ordifferent types of machine learning algorithms. In another embodiment,the inferred user attribute model includes different candidate datamodels that accepts different types of input variables to determinedifferent inferred user attributes. The inferred user attribute modelmay select a subset of the candidate data models for determininginferred user attributes, for a particular user, depending on theavailable input variables derived from the user information collectedfor the particular user. Additionally or alternatively, the inferreduser attribute model may select a subset of the candidate data modelsfor determining inferred user attributes, depending on the inferred userattributes needed.

In an embodiment, the machine learning system checks which inferred userattributes are required as input for determining whether to add the userto the inclusion group and/or exclusion group, as described above withreference to FIGS. 6A-B. In particular, the target group selectionsystem checks which attributes are to be evaluated using inclusionranges and/or inclusion weights at Operations 604 and 610. The targetgroup selection system checks which attributes are to be evaluated usingexclusion ranges and/or exclusion weights at Operations 624 and 630. Thetarget group selection system then generates one or more inferred userattribute models for determining the required inferred user attributes.The target group selection system may but does not necessarily generateany data models for determining other inferred user attributes.

In an embodiment, after generating an inferred user attribute model, themachine learning system stores the inferred user attribute model in adata repository. The machine learning system may subsequently obtain anupdated training set (including new historical user information and/ornew labeled inferred user attributes). The machine learning system maygenerate an updated inferred user attribute model by applying a machinelearning algorithm to the updated training set. Next time that theinferred user attribute model is needed, the target group selectionengine may retrieve the inferred user attribute model from the datarepository, rather than requesting the machine learning system togenerate an inferred user attribute model.

One or more embodiments include determining inferred user attributes byapplying the inferred user attribute model to the user information(Operation 708). The target group selection engine applies the inferreduser attribute model to the user information obtained at Operation 702to determine inferred user attributes.

Optionally, the target group selection engine manipulates the userinformation in a pre-processing phase. For example, the target groupselection engine may convert the user information into a standardizedformat. Additionally or alternatively, the target group selection enginemay encode the user information.

In an embodiment, the inferred user attribute model uses a regressionfunction. The target group selection engine inputs the user informationobtained at Operation 702 into the regression function. The regressionfunction outputs one or more inferred user attributes.

In an embodiment, the inferred user attribute model uses a knn datamodel. The knn data model includes sets of historical user information,associated with respective labeled inferred user attributes.

The target group selection engine determines a count k of a subset ofthe historical user information, associated with different users, to beselected for determining one or more inferred user attributes. Thetarget group selection engine also determines a difference measurementfunction. The difference measurement function is configured to determinea difference between (a) the k selected sets of historical userinformation, and (b) the user information obtained at Operation 702. Anexample of a difference measurement function is a function thatdetermines a Euclidean distance between two points.

Thereafter the target group selection engine selects k sets ofhistorical user information in the knn data model such that thedifference, determined by the difference measure function, is minimized.

The target group selection engine determines the inferred userattributes based on the k selected sets of historical user information,without using the remaining historical user information in the knn datamodel. The target group selection engine may determine an inferred userattribute by determining an average (such as, mean, median, or mode) ofthe labeled inferred user attributes associated with the k selected setsof historical user information.

As an example, a target group selection engine may determine that acount k of sets of historical user information to be selected is five.The target group selection engine may select five sets of historicaluser information in a knn data model that are nearest to a set of userinformation of a particular user. Two of the five sets of historicaluser information may be labeled with an inferred user attribute,“parent.” Three of the five sets of historical user information may belabeled with an inferred user attribute, “not a parent.” The targetgroup selection engine may determine an average associated with thelabeled inferred user attributes. In the above example, the target groupselection engine may determine that an inferred user attribute score forthe particular user for being a parent is ⅖=0.40.

In an embodiment, the inferred user attribute model uses a decisiontree. The target group selection engine identifies a variable associatedwith a first split of the decision tree. The target group selectionengine determines a value for the variable from the user informationobtained at Operation 702. The target group selection engine inputs thevalue as the variable associated with the first split to determine whichbranch of the decision tree to follow. The target group selection enginethen moves to a next split of the decision tree. The target groupselection engine iterates the above process until a leaf of the decisiontree is reached. The leaf determines one or more inferred userattributes for the particular user.

In an embodiment, the inferred user attribute model uses an ANN. Thetarget group selection engine inputs the user information obtained atOperation 702 into the ANN. The ANN outputs one or more inferred userattributes.

Additional and/or alternative functions and/or models may also be usedin the inferred user attribute model.

In an embodiment, the target group selection system checks whichinferred user attributes are required as input for determining whetherto add the user to the inclusion group and/or exclusion group, asdescribed above with reference to FIGS. 6A-B. In particular, the targetgroup selection system checks which attributes are to be evaluated usinginclusion ranges and/or inclusion weights at Operations 604 and 610. Thetarget group selection system checks which attributes are to beevaluated using exclusion ranges and/or exclusion weights at Operations624 and 630. The target group selection system then applies one or moreinferred user attribute models to the user information obtained atOperation 702 to determine the required inferred user attributes. Thetarget group selection system may but does not necessarily determine anyother inferred user attributes from the user information.

One or more embodiments include generating an affinity attribute modelvia machine learning (Operation 710). The machine learning systemgenerates an affinity attribute model via machine learning.

The machine learning system obtains a training set for training theaffinity attribute model. The training set includes user information andcorresponding labeled affinity attributes. The training set alsooptionally includes scores associated with the labeled affinityattributes. The scores indicate a degree to which a particular set ofuser information exhibits the labeled affinity attribute. Examples ofoperations for obtaining a training set are described above withreference to Operation 706.

As an example, a machine learning system may obtain the followingtraining set:

TABLE 5 Labeled Affinity User Information Attribute Name: Mary PositiveAffinity Towards: Birthday: Dec. 6, 1982 Professional clothing, scoreOccupation: Attorney 82 Web Interaction History: Visited Sportsclothing, score 76 sportsnews.com on Oct. 1, 2019; visited Technicalgear, score 60 investment.com on Oct. 2, 2019; visited Negative AffinityTowards: basketballteam.com on Oct. 2, 2019 Jeans, score 70 Name: JohnPositive Affinity Towards: Geographical Location: Los Angeles, CACookbooks, score 65 Education: Graduated UCLA in 2007 Negative AffinityTowards: Web Interaction History: Visited Romance books, score 76uclafootballteam.com on Oct. 3, 2019; visited restaurantabc.com on Oct.3, 2019

The machine learning system applies a machine learning algorithm to thetraining set to generate an affinity attribute model. Various machinelearning algorithms may be used. Examples of operations for applying amachine learning algorithm to a training set to generate one or moredata models are described above with reference to Operation 706.

One or more embodiments include determining affinity attributes byapplying the affinity attribute model to the user information (Operation712). The target group selection engine applies the affinity attributemodel to the user information obtained at Operation 702 to determineaffinity attributes. Examples of operations for applying one or moredata models to user information to determine attributes for theparticular user are described above with reference to Operation 708.

One or more embodiments include generating a user profile for theparticular user (Operation 714). The target group selection enginegenerates a user profile for the particular user. The user profileincludes (a) the confirmation user attributes determined at Operation704; (b) the inferred user attributes determines at Operation 708;and/or (c) the affinity attributes determined at Operation 712.

In an embodiment, the target group selection engine may iterate theoperations of FIG. 7 with respect to each set of user informationassociated with a different user. The target group selection engine maytherefore generate and/or enhance user profiles for multiple users.

6. Example Embodiments

Detailed examples are described below for purposes of clarity.Components and/or operations described below should be understood asspecific examples which may not be applicable to certain embodiments.Accordingly, components and/or operations described below should not beconstrued as limiting the scope of any of the claims.

FIGS. 8A-B illustrate an example user interface for specifying targetuser tuning parameters, in accordance with one or more embodiments.

As illustrated, user interface 800 is configured to accept specificationof inclusion tuning parameters. User interface 800 includes variousregions for presenting different types of information. One regionpresents campaign attributes 802. Other regions present confirmable userattributes 804, inferred user attributes 806, and affinity attributes808 relevant to determining inclusion of a user into a target group.

Campaign attributes 802 includes content attributes and productattributes. Content attributes are determined based on content itemsspecified in a campaign profile of a campaign. Product attributes aredetermined based on products specified in the campaign profile.

As illustrated, content attributes include “Clearance” and“End-of-Season.” The content attributes indicate that the campaign is aclearance and end-of-season sale. Content items may include, forexample, “End of Season Sale! Winter is ending and spring is here. Lastchance to snatch these deals,” or “Clearance Sale! Going, going, gone.Get these winter boots before it's too late.”

As illustrated, different types of product attributes may be availablefor different products. Products associated with the campaign includeItem #234. Product attributes include the product type “shirt,” theapplicable season for the clothing “winter,” the product brand “Niki.”Additionally, products associated with the campaign include Item #95.Product attributes include the product type “pants,” the applicableseason for the clothing “winter,” the product brand “Jane & Joe,” andthe material “all cotton.” Additionally, products associated with thecampaign include Item #750. Product attributes include the product type“dress,” the applicable season for the clothing “winter,” the productbrand “Niki,” and the dimensions “mid-length.”

An administrator enters, via user interface 800, user attributesrelevant for determining the inclusion group under confirmable userattributes 804, inferred user attributes 806, and/or affinity attributes808. As illustrated, the administrator has entered “Age,” “Female,” and“San Mateo” under confirmable user attributes 804. The administrator hasentered “Parent,” and “Working Professional” under inferred userattributes 806. The administrator has entered “Clearance,” “Niki,” and“Winter Clothing” under affinity attributes 808.

The administrator may use a variety of input methods for entering therelevant user attributes. The administrator may type in the userattributes. Additionally or alternatively, the administrator may selectthe user attributes from a dropdown menu. Additionally or alternatively,the administrator may drag-and-drop content attributes and/or productattributes from campaign attributes 802 into affinity attributes 808.

Additionally, the administrator enters, via user interface 800, (a)whether each relevant user attribute is to be evaluated using aninclusion range or an inclusion weight, and (b) a corresponding value.If an inclusion range is indicated, the corresponding value indicates arequired range of values for the user attribute for including the userinto the inclusion group. If an inclusion weight is indicated, thecorresponding value indicates a weight to be applied to a value for theuser attribute for determining inclusion of the user into the inclusiongroup.

As an example, the administrator has specified that the “Age” attributeis to be evaluated using an inclusion range. The administrator hasspecified that the corresponding value is “18-25.” The above inclusionspecification indicates that an inclusion range for “Age” is [18-25]. Ifa user is between 18 and 25 years old, the user is included in theinclusion group. Otherwise, the user is excluded from the inclusiongroup.

As another example, the administrator has specified that the “Female”attribute is to be evaluated using an inclusion weight. Theadministrator has specified that the corresponding value is “30%.” Theabove inclusion specification indicates that an inclusion weight of 30%is to be applied to the confirmable user attribute “Female.” Variousinclusion weights are applied to various user attributes to determine aweighted sum, which becomes an inclusion score. If the inclusion scoreis above a threshold value, the user is included in the inclusion group.Otherwise, the user is excluded from the inclusion group.

After submission of the inclusion specification, user interface 800 isconfigured to accept specification of exclusion tuning parameters. Userinterface 800 includes various regions for presenting different types ofinformation. One region presents campaign attributes 802. Other regionspresent confirmable user attributes 810, inferred user attributes 812,and affinity attributes 814 relevant to determining exclusion of a userfrom a target group.

An administrator enters, via user interface 800, user attributesrelevant for determining the exclusion group under confirmable userattributes 810, inferred user attributes 812, and/or affinity attributes814. As illustrated, the administrator has entered “Age,” “Book ShoppingHistory,” and “Los Angeles” under confirmable user attributes 810. Theadministrator has entered “Sports Lover,” and “Retiree” under inferreduser attributes 812. The administrator has entered “New Products,” and“Moko” (a product brand) under affinity attributes 814. As describedabove, the administrator may use a variety of input methods for enteringthe relevant user attributes.

Additionally, the administrator enters, via user interface 800, (a)whether each relevant user attribute is to be evaluated using anexclusion range or an exclusion weight, and (b) a corresponding value.If an exclusion range is indicated, the corresponding value indicates arequired range of values for the user attribute for including the userinto the exclusion group. If an exclusion weight is indicated, thecorresponding value indicates a weight to be applied to a value for theuser attribute for determining inclusion of the user into the exclusiongroup.

Based on the relevant user attributes, inclusion ranges, inclusionweights, exclusion ranges, and exclusion weights entered via the userinterface 800, a target group selection engine determines an inclusiongroup of users and an exclusion group of users. Users who are in theinclusion group but not the exclusion group are included in a targetgroup for receiving content items associated with the campaign. Otherusers do not receive content items associated with the campaign.

FIGS. 9A-C shows example Venn diagrams illustrating inclusion groups,exclusion groups, and target groups, in accordance with one or moreembodiments.

A target group selection engine has functionality to cause transmissionof content items to a set of users 900.

Referring to FIG. 9A, the target group selection engine appliesinclusion ranges and/or inclusion weights to user attributes of users900 to determine respective inclusion scores. The target group selectionengine compares each inclusion score with a threshold value. If aninclusion score for a particular user is above the threshold value, theparticular user is included in the inclusion group. Otherwise, theparticular user is not included in the inclusion group. The circles,inclusion group 902 and inclusion group 904, represent a subset of users900 for which the inclusion scores are above the threshold value.

Referring to FIG. 9B, the target group selection engine appliesexclusion ranges and/or exclusion weights to user attributes of users900 to determine respective exclusion scores. The target group selectionengine compares each exclusion score with a threshold value. If anexclusion score for a particular user is above the threshold value, theparticular user is included in the exclusion group. Otherwise, theparticular user is not included in the exclusion group. The circles,exclusion group 912, exclusion group 914, and exclusion group 916,represent a subset of users 900 for which the exclusion scores are abovethe threshold value.

Optionally, rather than evaluating all users 900, the target groupselection engine may evaluate only users of inclusion group 902 andinclusion group 904 to determine exclusion scores of only users ofinclusion group 902 and inclusion group 904. Selective evaluation fordetermining the exclusion group may increase efficiency for determiningthe target group.

Referring to FIG. 9C, the exclusion groups 912-916 are superimposed onthe inclusion groups 902-904 in a Venn diagram. Target group 920 isrepresented by the dotted-pattern regions. Users who are in any ofinclusion groups 902-904, without being in any of exclusion groups912-916 are within target group 920.

7. Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), or network processing units (NPUs)that are persistently programmed to perform the techniques, or mayinclude one or more general purpose hardware processors programmed toperform the techniques pursuant to program instructions in firmware,memory, other storage, or a combination. Such special-purpose computingdevices may also combine custom hard-wired logic, ASICs, FPGAs, or NPUswith custom programming to accomplish the techniques. Thespecial-purpose computing devices may be desktop computer systems,portable computer systems, handheld devices, networking devices or anyother device that incorporates hard-wired and/or program logic toimplement the techniques.

For example, FIG. 10 is a block diagram that illustrates a computersystem 1000 upon which an embodiment of the invention may beimplemented. Computer system 1000 includes a bus 1002 or othercommunication mechanism for communicating information, and a hardwareprocessor 1004 coupled with bus 1002 for processing information.Hardware processor 1004 may be, for example, a general purposemicroprocessor.

Computer system 1000 also includes a main memory 1006, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 1002for storing information and instructions to be executed by processor1004. Main memory 1006 also may be used for storing temporary variablesor other intermediate information during execution of instructions to beexecuted by processor 1004. Such instructions, when stored innon-transitory storage media accessible to processor 1004, rendercomputer system 1000 into a special-purpose machine that is customizedto perform the operations specified in the instructions.

Computer system 1000 further includes a read only memory (ROM) 1008 orother static storage device coupled to bus 1002 for storing staticinformation and instructions for processor 1004. A storage device 1010,such as a magnetic disk or optical disk, is provided and coupled to bus1002 for storing information and instructions.

Computer system 1000 may be coupled via bus 1002 to a display 1012, suchas a cathode ray tube (CRT), for displaying information to a computeruser. An input device 1014, including alphanumeric and other keys, iscoupled to bus 1002 for communicating information and command selectionsto processor 1004. Another type of user input device is cursor control1016, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1004 and for controlling cursor movement on display 1012. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1000 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1000 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1000 in response to processor 1004 executing one or moresequences of one or more instructions contained in main memory 1006.Such instructions may be read into main memory 1006 from another storagemedium, such as storage device 1010. Execution of the sequences ofinstructions contained in main memory 1006 causes processor 1004 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 1010.Volatile media includes dynamic memory, such as main memory 1006. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge,content-addressable memory (CAM), and ternary content-addressable memory(TCAM).

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 1002. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1004 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 1000 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 1002. Bus 1002 carries the data tomain memory 1006, from which processor 1004 retrieves and executes theinstructions. The instructions received by main memory 1006 mayoptionally be stored on storage device 1010 either before or afterexecution by processor 1004.

Computer system 1000 also includes a communication interface 1018coupled to bus 1002. Communication interface 1018 provides a two-waydata communication coupling to a network link 1020 that is connected toa local network 1022. For example, communication interface 1018 may bean integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example,communication interface 1018 may be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks may also be implemented. In any such implementation, communicationinterface 1018 sends and receives electrical, electromagnetic or opticalsignals that carry digital data streams representing various types ofinformation.

Network link 1020 typically provides data communication through one ormore networks to other data devices. For example, network link 1020 mayprovide a connection through local network 1022 to a host computer 1024or to data equipment operated by an Internet Service Provider (ISP)1026. ISP 1026 in turn provides data communication services through theworld wide packet data communication network now commonly referred to asthe “Internet” 1028. Local network 1022 and Internet 1028 both useelectrical, electromagnetic or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1020 and through communication interface 1018, which carrythe digital data to and from computer system 1000, are example forms oftransmission media.

Computer system 1000 can send messages and receive data, includingprogram code, through the network(s), network link 1020 andcommunication interface 1018. In the Internet example, a server 1030might transmit a requested code for an application program throughInternet 1028, ISP 1026, local network 1022 and communication interface1018.

The received code may be executed by processor 1004 as it is received,and/or stored in storage device 1010, or other non-volatile storage forlater execution.

8. Miscellaneous; Extensions

Embodiments are directed to a system with one or more devices thatinclude a hardware processor and that are configured to perform any ofthe operations described herein and/or recited in any of the claimsbelow.

In an embodiment, a non-transitory computer readable storage mediumcomprises instructions which, when executed by one or more hardwareprocessors, causes performance of any of the operations described hereinand/or recited in any of the claims.

Any combination of the features and functionalities described herein maybe used in accordance with one or more embodiments. In the foregoingspecification, embodiments have been described with reference tonumerous specific details that may vary from implementation toimplementation. The specification and drawings are, accordingly, to beregarded in an illustrative rather than a restrictive sense. The soleand exclusive indicator of the scope of the invention, and what isintended by the applicants to be the scope of the invention, is theliteral and equivalent scope of the set of claims that issue from thisapplication, in the specific form in which such claims issue, includingany subsequent correction.

What is claimed is:
 1. One or more non-transitory machine-readable mediastoring instructions which, when executed by one or more processors,cause: training, based on at least one training dataset, at least onemodel using machine learning to estimate positive affinities for a firstset of attributes and negative affinities for a second set of attributesbased on user information; applying the at least one model to a firstset of user information for a first user to estimate at least a firstvalue representing a positive affinity for at least a first attribute inthe first set of attributes and a second value representing a negativeaffinity for at least a second attribute in the second set ofattributes; determining, for the first user, an inclusion score based atleast in part on the first value representing the positive affinity forat least the first attribute and an exclusion score based at least inpart on the second value representing the negative affinity for at leastthe second attribute; determining whether the inclusion score satisfiesa first threshold and whether the exclusion score satisfies a secondthreshold; including the first user in a target group for receiving aset of content items, responsive at least to: determining that theinclusion score satisfies the first threshold and the exclusion scoredoes not satisfy the second threshold.
 2. The one or more non-transitorymachine-readable media of claim 1, wherein the inclusion score isfurther determined based at least in part on a set of inclusion scoreweights including a first inclusion score weight that is applied to thefirst attribute.
 3. The one or more non-transitory machine-readablemedia of claim 1, wherein the exclusion score is further determinedbased at least in part on a set of exclusion score weights including afirst exclusion score weight that is applied to the second attribute. 4.The one or more non-transitory machine-readable media of claim 1,further storing instructions which cause: receiving user inputspecifying target tuning parameters for tuning the inclusion score; andgenerating an inclusion score function based at least in part on thetarget tuning parameters; wherein the inclusion score is computed basedat least in part on the inclusion score function.
 5. The one or morenon-transitory machine-readable media of claim 4, wherein the inclusionscore function includes a set of inclusion weights that are applied tothe first set of attributes, wherein the inclusion score is furthercomputed as a function of the set of inclusion weights.
 6. The one ormore non-transitory machine-readable media of claim 1, further storinginstructions which cause: receiving user input specifying target tuningparameters for tuning the exclusion score; and generating an exclusionscore function based at least in part on the target tuning parameters;wherein the exclusion score is computed using the exclusion scorefunction.
 7. The one or more non-transitory machine-readable media ofclaim 6, wherein the exclusion score function includes a set ofexclusion weights that are applied to the second set of attributes,wherein the exclusion score is further computed as a function of the setof exclusion weights.
 8. The one or more non-transitory machine-readablemedia of claim 1, further storing instructions which cause: transmittingthe set of content items, associated with the campaign, to each userincluded in the target group.
 9. The one or more non-transitorymachine-readable media of claim 1, wherein the at least one modelincludes an inferred attribute model and an affinity attribute model,wherein the instructions further cause: applying the inferred attributemodel to the first set of user information to infer at least one userattribute and the affinity attribute model to estimate the first valuerepresenting the positive affinity for at least the first attribute andthe second value representing the negative affinity for at least thesecond attribute.
 10. The one or more non-transitory machine-readablemedia of claim 1, wherein the first threshold specifies an inclusionrange; wherein determining that the inclusion score satisfies the firstthreshold comprises determining that the inclusion score falls withinthe inclusion range.
 11. The one or more non-transitory machine-readablemedia of claim 1, wherein the second threshold specifies an exclusionrange; wherein determining that the exclusion score does not satisfy thesecond threshold comprises determining that the exclusion score fallsoutside of the exclusion range.
 12. The one or more non-transitorymachine-readable media of claim 1, wherein the first threshold specifiesa target value, wherein determining that the inclusion score satisfiesthe first threshold comprises determining that the inclusion score isabove the target value.
 13. The one or more non-transitorymachine-readable media of claim 1, wherein the second thresholdspecifies a target value; wherein determining that the exclusion scoredoes not satisfy the second threshold comprises determining that theexclusion score falls below the target value.
 14. The one or morenon-transitory machine-readable media of claim 1, further storinginstructions which cause: traversing a set of users that have inclusionscores satisfying the first threshold to determine whether the set ofusers are included in an exclusion group that have exclusion scoressatisfying the second threshold; and excluding at least one user thathas an inclusion score satisfying the first threshold responsive todetermining that the at least one user is included in the exclusiongroup.
 15. The one or more non-transitory machine-readable media ofclaim 1, further storing instructions which cause: receiving thetraining dataset from one or more data sources, wherein the trainingdataset includes labels that identify positive affinity values andnegative affinity value for a set of users; and training, using thetraining dataset, the at least one model to estimate positive affinityvalues for the first set of attributes and negative affinity value forthe second set of attributes for new users.
 16. The one or morenon-transitory machine-readable media of claim 15, wherein the trainingdataset includes historical user information obtained from at least oneof a web application or a cookie.
 17. The one or more non-transitorymachine-readable media of claim 1, wherein the at least one modelincludes at least one of a decision tree, an artificial neural network,a cluster model, or a support vector machine, wherein the at least onemodel is trained based at least in part on an error function thatdetermines a difference between inferred user attribute values andlabeled user attribute values.
 18. The one or more non-transitorymachine-readable media of claim 1, further storing instructions whichcause: receiving user input specifying target tuning parameters fortuning the inclusion score; and generating an inclusion score functionbased at least in part on the target tuning parameters; wherein theinclusion score is computed based at least in part on the inclusionscore function. receiving user input specifying target tuning parametersfor tuning the exclusion score; and generating an exclusion scorefunction based at least in part on the target tuning parameters; whereinthe exclusion score is computed using the exclusion score function. 19.A system comprising: one or more devices including one or more hardwareprocessors; and the system being configured to perform operationscomprising: training, based on at least one training dataset, at leastone model using machine learning to estimate positive affinities for afirst set of attributes and negative affinities for a second set ofattributes based on user information; applying the at least one model toa first set of user information for a first user to estimate at least afirst value representing a positive affinity for at least a firstattribute in the first set of attributes and a second value representinga negative affinity for at least a second attribute in the second set ofattributes; determining, for the first user, an inclusion score based atleast in part on the first value representing the positive affinity forat least the first attribute and an exclusion score based at least inpart on the second value representing the negative affinity for at leastthe second attribute; determining whether the inclusion score satisfiesa first threshold and whether the exclusion score satisfies a secondthreshold; including the first user in a target group for receiving aset of content items, responsive at least to: determining that theinclusion score satisfies the first threshold and the exclusion scoredoes not satisfy the second threshold.
 20. A method comprising:training, based on at least one training dataset, at least one modelusing machine learning to estimate positive affinities for a first setof attributes and negative affinities for a second set of attributesbased on user information; applying the at least one model to a firstset of user information for a first user to estimate at least a firstvalue representing a positive affinity for at least a first attribute inthe first set of attributes and a second value representing a negativeaffinity for at least a second attribute in the second set ofattributes; determining, for the first user, an inclusion score based atleast in part on the first value representing the positive affinity forat least the first attribute and an exclusion score based at least inpart on the second value representing the negative affinity for at leastthe second attribute; determining whether the inclusion score satisfiesa first threshold and whether the exclusion score satisfies a secondthreshold; including the first user in a target group for receiving aset of content items, responsive at least to: determining that theinclusion score satisfies the first threshold and the exclusion scoredoes not satisfy the second threshold.